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Abstract 

With the increased use of large-scale real-time embedded sensor networks, new control mecha- 
nisms are needed to avoid congestion and meet required Quality of Service (QoS) levels. In this 
paper, we propose a Markov Decision Problem (MDP) to prescribe an optimal query assignment 
strategy that achieves a trade-off between two QoS requirements: query response time and data 
validity. Query response time is the time that queries spend in the sensor network until they are 
solved. Data validity (freshness) indicates the time elapsed between data acquisition and query 
response and whether that time period exceeds a predefined tolerance. We assess the perfor- 
mance of the proposed model by means of a discrete event simulation. Compared with three 
other heuristics, derived from practical assignment strategies, the proposed policy performs bet- 
ter in terms of average assignment costs. Also in the case of real query traffic simulations, results 
show that the proposed policy achieves cost gains compared with the other heuristics considered. 
The results provide useful insight into deriving simple assignment strategies that can be easily 
used in practice. 
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1. Introduction 

Wireless sensor networks (WSN) are intended for sensing environmental phenomena and 
communicating the sensed data for further use. Applications of such sensor networks include 
forest fire detection, intruder detection and localization and indoor environmental control [1]. 
The increased computing capabilities of the modern sensor networks have enabled the WSNs to 
become an integrated platform, where local query processing is performed. Consequently, not 
only the Database (DB) is able to respond to queries on the sensed environment, but also the 
sensors within the WSN. However, an increased number of WSN queries poses scalability and 
Quality of Service (QoS) challenges. 

In recent years, studies in sensor networks focused on energy efficient data transmission |l2l[3] 
131 and the traffic was assumed to have unconstrained delivery requirements. However, growing 
interest in applications with specific QoS requirements has posed additional challenges. We refer 
to for an extensive outline of WSN specific QoS requirements. Literature reveals related 

work on QoS-based routing protocols within the sensor network. Most such protocols satisfy 
end-to-end packet delay [W] or data reliability requirements |7 , 8 1 or a trade-off between the two 
||9). Little work exists, however, on QoS guarantees in the field of sensor query monitoring, as 
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addressed in this paper. In flOl a query optimizer is used to satisfy query delay requirements. In 
IfTl I the authors use data validity restrictions to specify how much time is allowed to pass since 
the last sensor acquisition so that the sensors are not activated, but previous sensed data is used. 

This paper addresses the trade-off between two QoS requirements, namely query response 
time and data validity. Query response time is the time that queries spend in the WSN until 
they are solved. Data validity indicates the time since data stored in the DB was acquired from 
the WSN and whether that time exceeds a predefined tolerance. We propose a model in which 
response time requirements are met by ensuring timely delivery of the sensed data either directly 
from the sensors (WSN) or from a storage facility (DB), see Figure [T] Assigning all queries to 
the WSN leads to large response times. To prevent this, sensed data is reported to the DB by the 
sensors and queries are answered with reported data. In this case, the stored data provided to the 
query may exceed validity tolerances. We use Markov Decision Processes (MDP) to compute an 
optimal query assignment policy such that a trade-off between response time and data validity is 
achieved. 
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Figure 1: WSN seen as an integrated platform, wliere queries generated by the end-users can be solved either by the 
WSN or by the DB 



The remainder of this paper is structured as follows. In Section|2]we describe the model of the 
query assignment problem and define it as a Discrete Time MDP. In Section[3]numerical results 
on the performance of the optimal policy in comparison to other feasible heuristic are presented. 
Concluding remarks and an outline of future research directions are provided in Section]?] 



2. Model Formulation 

2.1. Model Description 

Consider a system consisting of a service facility (WSN) with processor sharing capabilities 
and a storage facility (DB). Figure ]2] shows the proposed model. The processor sharing type of 
service assumed for the WSN reflects the lEE 802. 15.4 MAC design principle of distributing the 
processing capacity fairly among the jobs simultaneously present in the network. Processor shar- 
ing service discipline for WLAN is validated by simulation in IIT2II . The same service discipline 
is assumed in [ 1 1 1 for query processing in WSNs. 

Two types of jobs, queries and reports, arrive at the system according to a Poisson process. 
Queries arrive with rate X\. Reports arrive with rate X2. The service requirements of the jobs are 
exponentially distributed with parameter /i, independent of the job type. 

After a report is processed, the sensed data is sent to the DB for long-term storage. Incoming 
queries are handled by a central query controller which can assign the query either to the DB or 
to the WSN. When assigned to the DB, queries are immediately answered with stored data. If 
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Figure 2: Proposed model with a central controller (C ), the database (DB) and the wireless sensor network (WSN). 



assigned to the WSN, queries wait to receive the sensed data, sharing the service with the other 
jobs present in the network. Therefore, the assignment decision is based on the trade-off between 
the two aforementioned QoS requirements. Our goal is to find an optimal assignment policy that 
achieves a trades-off between query response time and data validity. 



2.2. Stochastic Dynamic Programming Formulation 

In this section, we formulate the assignment problem as a Markov Decision Problem. In 
subsection |2.2.1[ we consider a Continuous Time Markov Decision Process (CTMDP) with a 
drift. For computational reasons, we construct an exponentially uniformized Markov Decision 



Process in subsection 2.2.2 This leads to the formulation of the assignment problem as a Discrete 
Time Markov Decision Problem in subsection 1223] 



2.2.7. Continuous Time Markov Decision Process with a Drift 

Consider the following Continuous Time Markov Decision Process (CTMDP) with a drift, 

• State space S = No x No x (0, 00), where (/, j, t) e 5 denotes the state in which there are / 
queries, j reports and t the time since last report completion. 

• Action: when the system is in state (;, j, t) e S , the controller takes an action from the 
action space D - {DB, WSN}, where d - DB denotes a DB assignment and d - WSN 
denotes a WSN assignment. 

• The transition rates, when in state (/, j, N) e S and action d e Dis taken, are as follows. 
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where j) = r-p, 02(i, j) - ^ due to the Processor Sharing service discipline assumed for the 
WSN. The first and second line of the expression model a query arrival under action d. The third 
line models a report arrival. The last two lines model a query and report completion, respectively. 
In addition, there is a deterministic drift for the age component t, which linearly increases as long 
as no report is completed [13.1 . 
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Formally, the dynamics of this controlled Markovian decision process is uniquely determined 
by its infinitesimal generators, |J4,| . For the described system under action d, this generator is 
specified, for any arbitrary function / : 5 x 5 x (0, oo) — > M, as follows, 

A'fd, j, f) = ■ + j/^'' j' 

(ij,iy 

The generator shows that a jump to a new state (/, j, t)' occurs with rate or no jump occurs, but 
the time evolves. 

• Cost: when in state (/, j, t), it is incurred a cost rate / for the queries waiting in the WSN 
and an instantaneous cost (f - TY, - max(x, 0) upon a DB assignment. 



2.2.2. Exponentially Uniformized Markov Decision Process 



The assignment model is described in subsection 2.2.1 by a Markov Decision Process with 
a drift. Alternatively, we could use a time discretization approach as in |i5|. However, both 
approaches would lead to technical weak convergence results and no computational results can 
be obtained directly as the process would have a continuous state component. For computational 
purposes, therefore, we artificially construct an exponentially uniformized Markov Decision Pro- 
cess, which will lead to both a discrete time and a discrete state MDP, as follows. 

Let B an arbitrary large finite number with B > X\ + X-i l^- At exponential times with 
parameter B, the system will have a transition. Denote by s the exponential realization of a 



transition. Given the state space assumed in subsection 2.2.1 the transition probabilities under 
action li e D are as follows. 
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It can be shown that the infinitesimal generators of this exponentially uniformized Markov 
Decision Process are the same as for the original Continuous Time Markov Decision Process 
(subsection 2.2.\) . Therefore, the constructed process and the original one are stochastically 
equivalent (see lfT4l ). 

Now observe that the time component t of the state space (/, j, t) becomes essentially a suc- 
cession of exponential phases. As a Markov Process, therefore, it is sufficient to keep track of the 
number of exponential phases A^, instead of t, the time since last report completion. The number 
of exponential phases approximates the time until a report completion hy t + s - {N + Y) ■ B^"^ . 

We can now restrict ourselves to a discrete state space Markov Decision Problem, with S - 
No X No X No, where (;, j, t) € S denotes the state in which there are / queries, j reports and 
steps since last report completion. 
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2.2.3. Discrete Time Markov Decision Problem 

Based on the exponentially uniformized model in Section 2.2.2 we formulate our assignment 
problem as a Discrete Time Markov Decision Problem (DTMDP) as follows, 

• State space: S - x Nq x Nq, where (i,j,N) € S denotes the state where there are ; 
queries and j reports in the WSN and the age of the stored data is A^, with the number 
of steps (exponentially distributed with the uniformization parameter) since the last report 
completion. 

• Action space: When in state (/, j, N), the query controller takes an action from the action 
space D = {DB, WSN}, where d - DB denotes a DB assignment and d - WSN denotes a 
WSN assignment. 

• Transition probabilities, when the system is in state (/, j, N) e S and action d e Dis taken, 
are as follows. 
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AjB e {1,2) and ^' - jjB as per uni- 



formization (see Section 2.2.2 1. The first two lines of the expressions model query arrivals 
under action d € {WSN, DB}. The third line models report arrivals. The forth and fifth line 
model query and report completions, respectively. The sixth line is a dummy transition as 
result of the uniformization. The last line prohibits any other state transition. Notice that 
every step, the age is incremented and at a report completion, the age is set to zero. 

• Cost function: The system incurs a cost ; for the queries waiting in the WSN. This can 
be interpreted as, each step ( of expected length B '), the system 'pays' one unit for each 
waiting query. At the end of a query's service, the system would had payed one unit 
for each step the query was in the system, i.e. the query response time. If a query is 
assigned to the DB, an instantaneously penalty C - max{N - T)'^,(x)^ = max{0, x), is 
incurred for exceeding the validity tolerance of the stored data, where T is a predefined 
time tolerance. In this case, the system 'pays' one unit for each step the data validity is 
exceeded. Therefore, when the system is in state (/, j, N), the cost incurred per step is: 

C(i,j,N) = / + A\(N - Tyi(d=DB), where (jcf = max{0, jc). 
Now, the quadruple {S, D, P, C) completely describes the DTMDP. 

To determine an optimal assignment policy and to use standard dynamic programming, define 



V„(/, j, N) : - minimal expected assignment cost over n steps starting in state(/, j, N). 
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Then V„(/, j, N) can be computed recursively by means of the value iteration algorithm (see lfT6l 
Section 8.5. 1 ). Consider V()(/, j, N) = Q and use the following backward recursive equation, 



/ + /I'j min 



V„(i+lJ,N+l) 
(N-rr + V„(i,j,N+l) 



(1) 



+4y„(/,7 + i,A? + i) 

+fi'(pi(i,j)Vn(i -IJ,N+ l)l,>o 
+fi'^2{iJ)V„{i,j -l,Q)lj>o 

- + A'^+n'lnj>o)]Vr,(i,j,N+ 1). 

The first term of the right-hand side of ([TJ is the cost of having / queries in service and a query 
assignment either to the WSN or to the DB. The next three terms represent the cost incurred 
by a transition due to a report arrival, a query completion and a report completion, respectively. 
Finally, the last term is the dummy term due to uniformization. 

Simultaneously to computing 'V„(i,j,N), the algorithm computes a e-optimal stationary pol- 
icy 7T„ which associates an optimizing action to the right-hand side of ([TJ for any state (i,j,N). 
Given the assignment policy, it is possible to compute the average assignment cost. Denote the 
minimal average assignment cost by g* = lim„^oa[V„+](i, j, N) - V„(i,j,N)], ifTTl . Since the un- 
derlying Markov chain is ergodic, g* is independent of the initial state. We approximate g* using 
the following bounds introduced in LlTil . 



L' < 2* < L'' where 



L'„ = mm[Vn^i{i,j,N) - V„{Uj,N)], L'^ = max[y„+i(/,;,A?) - Vn{i,j,N)]. 

More precisely, the optimal cost g* is computed with an accuracy e by iterating the right-hand 
side of ^ until L"„ - L,', < e. Then the average assignment cost is approximated as g* ~ 
(L"„ + L',)/2. It can be shown that the lower and upper bound converge in a finite number of 
steps (Theorem 8.5.4 ||16J ) to the optimal cost. 

3. Simulation Results 

In order to illustrate the performance of the optimal policy, we present the associated average 
cost (g*), as defined in Section |2] for different tolerances (r). We compare the performance of 
the proposed assignment policy with several fixed heuristic policies. 

Figure [3] shows what action is the optimal policy associating with a state {i,j,N), when dif- 
ferent validity tolerances are assumed. 

3.1. Fixed Heuristics Policies for Performance Comparison 

In order to further numerically analyze the performance of the proposed assignment strategy, 
we consider the following three fixed heuristic policies: 

• A fixed heuristic policy tt^* that always assigns incoming queries to the DB. Upon a query 
arrival, the cost incurred is (A^ - T)^. 

• A fixed heuristic policy that always assigns incoming queries to the WSN. 
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Figure 3: WSN assignment (black) and DB assignment (grey) assuming Ai = 0.8, A2 = 0.5 and p = 1.8. 



• A heuristic policy that always assigns incoming queries to the DB if the age does not 
exceed the tolerance, i.e. <= T, and to the WSN otherwise. 

Theorem 1. Assuming the DTMDP parameters ^[,^'2 ond ji' , the average assignment cost of the 
heuristics tt^* and are as follows , 

-4)^+1 

C„Db - (2) 

C„w = y (3) 

fi' - (i + i 



Proof. Appendix A □ 



Simulation results show that, compared with the heuristics, the optimal policy achieves a 



lower average assignment cost (Figure 4(a) 1. The cost difference is significant for small time 
tolerances. This is of particular interest for real-time applications which specify low time toler- 
ances. In the limit, r — > 00, both 7:^ and ;7r^* approach the optimal policy. This can be explained 
by the fact that the stored data is considered valid for a longer time. Consequently, DB assign- 



ments under n^, tt'^'' and tt'^'" become more frequent (Figure 4(b) 1 since they result in small or 
no penalties at all. The load of the WSN is, therefore, considerably decreased. 
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Figure 4: Average Assignment Cost and DB utilization assuming query and report airival rates Ai = 0.8 and A2 = 0.5, 
respectively and WSN service rate /j = 1.8 



Also in the case of query arrival increase (Figures 5(a) and 5(b) 1 or processing capabilities 
increase, the optimal policy outperforms the heuristic policies in terms of average assignment 
costs (Figures 6(a) and 6(a) 1. 




Figure 5: Average Assignment Cost for different query arrival rates Ai, A2 = 0.5, /J = 1.8 and T = 1 

Such insight into the performance of the system enables WSN service providers to deliver 
customized and efficient monitoring services to the end-users. For reasonably large data validity 
tolerances, simple heuristics such as n'^'' or perform well in comparison to the optimal policy. 
These are particularly suited for monitoring environments with little variation over time, e.g. 
temperature sensing in forests. 

For appUcations with highly constrained delivery requirements and large data variance over 
time, such as fire detection or CO2 monitoring, however, our proposed model outperforms the 
heuristics. Moreover, as seen in Figure [3] the optimal policy assigns incoming queries to the 
WSN only if the number of reports in service exceeds the number of queries. A large number of 
reports in service ensures frequent DB updates which, in turn, decrease the assignment costs. 
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Figure 6: Average Assignment Cost for different processing capabilities A[ = 0.8, A2 = 0.5 and T = I 



3.2. Policy Simulations for Real Query Traffic 

In this subsection we assess the performance of the above described policies using data ob- 
tained from a commercial sensor network platform fTSl. We use a log file containing timestamps 
(in seconds) of the queries arriving at the platform. We selected two time periods, depicted in 



Figures 7(a) and 7(b) which are representative for the intensity of query arrivals. 

Dataset 1 contains timestamps of queries from one weekday around lunch time, a period 
when the platform typically receives many queries (see Figure 7(a) i. The number of queries 



arriving at the platform per minute varies from about 25 in a busy period, to 5 or less in a quiet 
period. The coexistence of such busy periods and quiet periods makes the assumption about 
query aiTivals originating from a homogeneous Poisson process invalid. Hence, verifying the 
performance of our optimal policy on such non-Poisson data provides valuable insight into its 
practical relevance. 

Dataset 2 has timestamps of queries arriving at the platform during a night (01:00 until 



07:30), see Figure 7(b) In this timespan, most queries are generated by a periodically refreshing 
dashboard and show little variance. 
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(a) Dataset 1 (b) Dataset 2 

Figure 7: The number of arriving queries per minute for the two datasets. 



We perform a discrete event simulation and use the timestamps from the datasets as arrival 
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moments of the queries. The optimal policy is determined using the procedure outlined in Section 
|2] The query arrival rate, Ai, is estimated from the mean interarrival time of the queries in the 
datasets. We choose the report arrival rate A2 and the service rate ju such that the system has the 



same load as the one in Figure 4(a) 




(a) Average Assignment costs (b) DB utilization 



Figure 8: Average Assignment costs and DB utilization for Dataset 1 



For Dataset 1, Figures 8(a) and 8(b) show that our optimal policy is better than the fixed 
heuristics in terms of average assignment costs. The load of the WSN is considerably decreased 
by routing the queries to the DB. The difference in performance is especially visible for smaller 
time tolerances, where the optimal policy achieves lower average costs whilst making more use 
of the DB. Results are similar for Dataset 2, see Figures 9(a) and 9(b) The optimal policy shows 
performs better than the fixed heuristics in terms of average assignment costs. Simulation results 
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(a) Average Assignment costs (b) DB utilization 

Figure 9: Average Assignment costs and DB utilization for Dataset 2 

show that the optimal policy achieves cost gains independent of the assumption that the arrivals 
at the platform follow a Poisson process. These results emphasize the applicability in practice of 
our proposed model. Lastly, we point out that the proposed model is independent of the WSN 
platform and its applications. 
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4. Conclusion and Future work 



In this paper we investigated the trade-off between the query response time and the validity 
of the stored data. Firstly, we defined a Continuous Time Markov Decision Process to balance 
between the query response time and the validity of the stored data provided. For computational 
reasons, we restricted ourselves to a Discrete Time Markov Decision Problem. Given specific 
data validity tolerances, we provide a query assignment strategy such that the query response 
time is minimized. Secondly, we assessed the performance of the proposed assignment strategy 
in comparison to several heuristic policies by means of discrete event simulations. For low data 
validity tolerance, the proposed policy is shown to achieve significant cost gains in comparison 
to several feasible heuristics. The proposed assignment strategy outperforms the heuristics also 
in the case of a representative WSN platform. Future work includes enhancing the query assign- 
ment model to incorporate additional metrics such as reporting rate, reliability of the sensors, 
accuracy of the sensed data and energy efficiency. 
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number 1GC1020, supported by the Subsidieregeling Sterktes in Innovatie. 
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Appendix A. Proof of Theorem [T] 

Proof. The policy is independent of the validity tolerance. The WSN behaves as a regular 
M/M/1 PS queue. Hence, the cost of the heuristic is given by the expected number of jobs in the 
WSN. 

C„w = E(0 (A.l) 

_ M M + M 
Ai+ A2 11- {Ai + /I2) 

H-{Ai+ A2) 

We define the cost of the policy tt^* in terms of the limiting probabilities as follows, 

= ^1 ^ TiNiN) ■ (N - T)\ (A.2) 

N>T 

where 7Tn(N) - 2 n(j, N) is the long run proportion of time that the process is in state A^. 

j 

We have the following balance equations for component j, 

'nj(0)^finjil) + (l-A2)7Tj(0) 

. nj(l) = fi7Tj(2) + {l-A2- fi)nj{l) + iz^r/O) (a.3) 

= 1 

V k 
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where ;r/0) = 2<0,A^). 

N 

Solving (|A.3|l, we have that 



Notice that 



Now 



n{0,N) = (1 - A2-iu)7t(0,N- 1) + iin{<d,N - 1) 
= (1 -i2f;r(0,0) 

;r/0) = 2;r(0,A^) 

N 

= 2^1-^2^(0,0) 



= -^(0,0) 

/t2 



From (|A.61l and (|A.4|), we have that 



7r(0,0) 



A* 



We have the following balance equations for component A^, 



;r^(0) = (1 - ^i)Tit,{^ - 1) + A^7r^(0, A? - 1) 

^A'(O) = ju 2 7r(A^) -piY. 7r(0, A?), with 7r(A?) = ^ 7r(;; A?) 

i; TTivC^) = 1 
V k 



where tt^^CA^) - 2 A^)- 



But 2 7r(A^) = 1 and 2 7r(0, A^) = tt/O) = f 7r(0, 0) as per ( A.6 1. 

N A- ^ I 

Now (|A.8|l become. 



;rM(0) = (1 -//);r^(A?- l)+//7r«(0,A?- 1) 

;r^(0)=//[l-i;r(0,0)] 

2 ;rAr(^) = 1 



V k 



Solving for ( A. 9 1, we have that 



n^iN) = ^2(1 - Aif 



We can now compute the cost ( A. 2 1 as follows. 
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C^.=AiYj^NiN)-iN-Tr 

N>T 
N>T 

= AiJ]A2a-A2f'^^-N' 

N'>0 

= AiA2il - A2f Yj^I - A2f' ■ N' 

N'>0 

= AiA2il - A2f^' 2(1 - A2f'-' ■ N' 



N'>0 

= Aa2(i-A2f^\-^y 
A2 

A2 



□ 
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